During the past 10 years, multimedia technology has undergone a dramatic change:
from analog to digital. A digital media compression technique has been applied to various user products including digital TVs, DVDs and MP3's, etc, and has reduced a distance between a user and a contents provider by employing international standards, MPEG-1, MPEG-2 and MPEG-4, to the digital media compression technique. In particular, the MPEG-4, started standardization from 1993, has introduced a graphic object compression conception and has become general media of a new concept.
Recently, applications using three-dimensional video are gaining high spotlight. Three-dimensional video is classified into three categories: a panoramic video, a multi-view video and a stereo video. The stereo video is purposed to present stereo effect by providing different images to the left and right eyes of a viewer. The multi-view video provides different views of an object or scene by capturing at different view directions. The panoramic video, which also requires multiple capturing of scene, provides wider angle of a scene by combining multiple images of adjacent scenes.
FIG. 1 illustrates a method of obtaining three-dimensional video. Referring to FIG. 1, images obtained from multiple cameras using parallel or divergent views are combined to generate panoramic image data, whereas images from the convergent view are used for multi-view image data.
FIG. 2 illustrates a general process of panoramic video handling. Referring to FIG. 2, the process includes an acquirement step, a processing step and a displaying step. The acquirement step acquires a plurality of images using a single camera or multiple cameras. The acquired multiple images are synthesized into a single image and this process is called a registration. Herein, a characteristic value of a camera is an important factor in the registration. The synthesized image is encoded, transmitted, and stored during the processing step. A reproduced panoramic video is displayed on a planar or nonplanar display.
Important factors in the acquirement step for acquiring the panoramic video include the number of cameras used for obtaining images, three-dimensional geometry and orientation information on each camera. These factors are essential for obtaining high quality panoramic image.
FIG. 3 shows an example of a panoramic picture reproduced on a cylindrical display. The display can be designed as simple as a cylinder or a sphere, and it can be designed more complicated like as a polygonal mesh. A two-dimensional map is still used to represent pixels in a cylinder or a sphere. Accordingly, it is effective that a cylinder or sphere is constructed by combining multiple planes as patches shown in FIG. 3, in terms of the quantity of calculations.
Without regarding to the shapes of display, a panoramic picture requires a larger capacity compared to a single-view picture. Thus, efficient compression and video data streaming are important.
When acquisition and synthesis of images for generating a panoramic image are finished, the panoramic image is encoded so as to store or transmit. The panoramic video compression is not much different from conventional video compression, except for information on a polygonal model on which the panoramic image is displayed. However, a fundamental difference between the panoramic video and a general video is that the panoramic video requires a larger capacity and higher picture quality compared to the general video.
In the case of the panoramic video, a ‘random access’ functionality is an important factor together with high compression efficiency. The random access function enables to directly access a desired frame or region in an image sequence.
A general method of video compression includes an interframe coding method that encodes a current frame using data predicted from a previous frame in order to remove temporal redundancy between similar pictures in sequence. The interframe coding method increases compression efficiency but it is not advantageous in terms of random access.
A video system that requires random access should accept delay time corresponding to at least several frames in order to access an interframe-coded image. In particular, in the case of the panoramic video, the delay time represents a vast amount of calculations and large memory capacity because the panoramic video has high resolution. In other words, the panoramic video has a problem that compression efficiency and system complexity collide with each other.
Although the interframe coding method is replaced with an intraframe coding method in order to avoid the delay time corresponding to several frames, the delay time still remains depending on circumstances, for example, a period of time required for decoding a single frame can be problematical. A period of time required for decoding a single frame of a panoramic video having the size of 4K×4K is estimated to be 120 times a period of time required for decoding a frame in SIF (352×240) format. Accordingly, real-time decoding may be difficult to be implemented since this calculation may take too much time even if a plurality of decoders is simultaneously employed thereto.
FIG. 4 illustrates an example of reproduction of a local area through a partial decoding in random access. Unless a special method is designed for local region decoding, the whole panoramic image must be decoded first even it is displaying a part of the whole image, and it is wasting system resources.
For example, let's assume the case of encoding a panoramic image by the conventional compression techniques such as JPEG, JPEG1200, MPEG4 Video SP Intra coding, MPEG4 Video SP Inter coding, MPEG4 AVC Intra coding and MPEG4 AVC Inter coding. Because these techniques were designed without considering the functionality of local region decoding, when we display the panoramic image on a smaller display screen, the whole image must be decoded first, then stored in the memory of devices, and finally the region data is sent to display.
Therefore, no aforementioned codec satisfies both the compression efficiency and the random access functionality. Thus, a new compression method satisfying these two functions is required to effectively compress the panoramic video.